When British tennis champion Fred Perry became the first player to win a career grand slam in 1935, he might have hoped his legacy would be defined by the stunning bit of history he made, still just 26 years old. It’s unlikely he could have predicted his name would be used in 2020 to uniform a far-right male militia jacked up on violence and misogyny. And yet, Proud Boys, an organisation allegedly founded as a joke by Gavin McInnes in the run-up the 2016 US election, has become instantly recognisable by its allegiance to Fred Perry’s black and yellow trim polo, forcing the brand to publicly distance itself and announce last week that it had withdrawn sales of the shirt in the US and Canada a year ago.

McInnes, 50, is the Scottish-Canadian co-founder of Vice Media, and lives in Brooklyn. He believes western culture is under siege and that feminism is a cancer. His group, much like an enraged Reddit sub-forum given vein-popping physical form, has been described as an alt-right fight club and hate group by Southern Poverty Law Centre (SPLC), as white supremacists by Joe Biden,and classified as an extremist group by the FBI – even though McInnes rejects the notion that Proud Boys are racists . It is not, by a long stretch, a good look for Fred Perry.

“It is incredibly frustrating that this group has appropriated our black and yellow twin-tipped shirt and subverted our laurel wreath to their own ends,” the company said on its website last week.

The statement added: “We are proud of its lineage and what the laurel wreath has represented for over 65 years: inclusivity, diversity and independence.” Fred Perry is unequivocal that it has “absolutely nothing” to do with Proud Boys and that “that association is something we must do our best to end”.

That embroidered circular flick, modelled on the original Wimbledon logo, has been adopted by many subcultures since the first polo was launched in 1952.

“It is easy for the piece to be taken up as uniform because it is designed to look like one. It is stern and sensible and needs a distinctive kind of look to pull it off. That look is not a very large and very aggressive and very pink man,” said fashion writer Tony Glenville.

In May, Fred Perry launched a new line with a publicity shoot featuring only models of colour. In pure Twitter bait, fury and counter-fury spewed online as some white customers claimed they would boycott the brand for “spreading diversity bollocks”. In a statement to Dazed and Confused magazine, Fred Perry replied: “We believe actions speak louder than words … Our real fans know what we stand for, and their response to this speaks volumes.”

It is not the first time the brand has been fashionable for groups on the fringes of society; part of the appeal of a neat, utilitarian Fred Perry polo is that it is subversively nonconformist. There is an ironic fashion joke at play: the aesthetic might look objectively square, but its spirit is rebellious.

Few brands have been tussled over as hard by competing subcultures. From tennis nuts to Jamaican rudeboys, skinheads, mods, ska-punks, indie kids and Camden popstars, all have done the Perry polo before Proud Boys came along. The brand has been worn by racist skinheads before McInnes’s lot and, despite its current wobble, is certain to be worn by music fans for some time still.

Amy Winehouse sported hers all over London and ended up collaborating with the brand in 2011 on a collection of Perry classics with Winehouse twists (collars turned up, sleeves capped in semi-sheer fabric). The line still sells well, particularly in the Far East. Damon Albarn, Britpop’s poster boy for the knitted cotton Perry pique shirt, was able to request a specific style of eight shirts which he wore for Blur’s reunion gigs in 2009.

Musicians have been essential to the brand’s credibility, be it the Specials and the Jam or Arctic Monkeys and Skepta. In a project for the brand’s 60th anniversary in 2012, Don Letts made a series of films tracing the line of cultural scenes and musical hierarchies that emerged in Britain since the teddy boys of the 1950s. Fashion was key, but it’s intriguing to see how little the look has shifted for Perryheads, whether they’re on scooters revving around Southend in the 60s or dancing at the 100 Club in the 90s.

Perry was the son of a textile factory worker born in Stockport. He first became a world table tennis champion at 19 before going on to win three consecutive Wimbledon titles. Despite his record-breaking success, he was treated with contempt by the elite who ran the sport in Britain. To Andrew Groves, professor of fashion design at Westminster University, it is this contrast between Perry’s underdog status and unquestionable personal glamour that has helped define the brand.

“The working-class authenticity of both Fred Perry the man and Fred Perry the brand allows it to resonate with each new generation,” he said. “Its no-nonsense design has enabled it to be reinterpreted by each emerging subculture in a way that gives it additional layered, and sometimes contradictory, meanings. Fred Perry was worn on the terraces at Chelsea but also in the gay bars on Old Compton Street; by skinheads at NF rallies but also by Jamaican rudeboys.”

Perry was a heartthrob: he dated Hollywood actresses, including Marlene Dietrich, and married four times. He moved to the US and took up citizenship there before launching his sportswear line with Australian footballer Tibby Wegner in the late 1940s. The company was kept in the family until Perry died in 1995, when it was bought by Japanese company Hit Union.

Groves believes the brand has been able to transcend each decade because of the way it has been reinterpreted by new fashion tribes. “It’s ironic therefore to see this particular shirt adopted by the Proud Boys,” he said, “given that within gay culture, a black polo shirt with yellow tipping on the collar usually signifies that the wearer is into watersports.”

What Fred Perry would think about all the symbolism at play on his bestselling shirts is another matter.

New BalanceIn the early 00s, the athletics brand was adopted by neo-Nazi groups in Germany. The company clawed back credibility by sponsoring anti-racist music events.

BurberryThe Burberry check, whether real or counterfeit, became so synonymous with football hooligans and thugs that in 2004, two Leicester pubs banned any customer from wearing the pattern. It was a long crawl back to its contemporary reinvention as a super luxury British brand.

Stone IslandIn the mid-1980s, the then obscure Italian sailing label became huge on the terraces with football casuals, who matched designer clothes with lairiness. The look went “weekend suburban dad” after some time, but the brand was given a surprising new twist when the rapper Drake began wearing it.